Joint Audio-Visual Unit Selection – the JAVUS Speech Synthesizer

The author presents a system for speech synthesis that selects and concatenates speech segments (units) of various size from an adequately prepared audio-visual speech database. The audio and the video track of selected segments are used together in concatenation to preserve audio-visual correlations. The input text is converted into a target phone chain and the database is searched for appropriate segments representing sub-chains of at least two phones that can be concatenated to the target utterance. The final segment sequence is selected from the possible segment sequences by a weighted sum of concatenation criteria for the audio and the video join. The weights of these audio and video join costs can be used to trade off between fluency in the audio and the video channel of the synthesized speech. The output shows the input text audio-visually spoken where the audio and the video track are reasonably fluent, synchronous, and intelligible.